Conversation
…profiling as part of our CI/CD pipeline
…rt data during load testing
* hotfix for missing validation (#577) * Updating oscars-backend service to support nullable projectId property * Feedback updates * Updating tests to cover NsiService with new Connection.projectId field * Cleaning up HoldController imports. Updating HoldControllerSteps to include a test with a projectId field when placing a connection hold * Ensuring the NsiService.hold() method only attempts to process the projectId field if it's not null, and not empty * add CORS origin customization property (#581) * fix: [OS-604] yang patch serialization (#582) * fix serialization error * more serialization fixes update spring boot and other libs * add auth interceptor to patch client * remove baseUrl from restClient --------- Co-authored-by: Alejandro Albino <aalbino@lbl.gov> Co-authored-by: Alex Albino <webmaster@alexventure.com>
…ut projectId field
…ults in a null projectId
feat: [OS-586] add project id to NSI API
* add project id set & view in frontend adjust hold delay minor fix for standalone CORS config * readjust hold delay
ci: [OS-571] Integrate OSCARS backend load testing solution with CI/CD pipeline
…or reserve() state changes. Refactoring NsiAsyncQueue.processQueue() task into testable components. Updating the NsiProvider cuke feature and testing steps to work with async processQueue() task. Adding new AsyncCallback class to work with async callback handlers.
…it story scope (revised AC)
test: [OS-484] NSI reserve, RESERVE_CHECKING -> RESERVE_HELD
…ys (#593) * make sure NSO dismantles happen before builds / redeploys * add tests around queue preprocessing
* projectId now a set instead of a single string * projectId now projectIds plural * update ESE controller tests to use real OSCARS logic (#594) * projectId now a set instead of a single string * projectId now projectIds plural * correct has / does not have a projectId checks
(the pipeline failed for the release branch since backend:profile has a rule not to run then, but it was a non-optional needs dependency for backend:push)
* add a python database report tool
…graphql data (#600) * add projectIds field validation against ESDB graphql data * add orcid check digit validation
* fix southbound queue behavior
…605) * Fix database error when committing a modify (OS-652) Add ScheduledTaskConfig bean to tell the Spring TaskScheduler to use more than one thread Rename `nso.queue-interval-millisec` to `nsi.queue-delay` * add sb.trigger-delay to control southbound trigger frequency * separate slow NSI housekeeping from other scheduled threads add resv.state-delay and nsi.housekeeping-delay props * fix NSI provider tests
* Handle rare NPE during a modify commit Ensure triggerSouthbound() finally releases connection lock Fix vlan validation Clean up some logging Don't throw exception when receiving multiple terminate requests, just succeed * fix determining port encapsulation * update esnet topology document * more gracefully handle errors
haniotak
added a commit
that referenced
this pull request
May 7, 2026
* develop to master (#609) * develop to master (#610) * 1.2.36 release (#615) * 1.2.37 release (#625) * 1.2.38 release (#631) * test (#632) * Current work on OSCARS backend profiling setup * Current work on PoC work for CI/CD profiling setup * PoC complete, we can now demonstrate how to integrate oscars backend profiling as part of our CI/CD pipeline * Adding PROFILING.md documentation * Updates to documentation * Fix markdown syntax mistake * Adding initial OSCARS backend load test container setup * Updates to load testing plan and docker file setup * Updating docker config to include a volume for the generated web report data during load testing * Ignore load testing generated volume * merge master to develop (#583) * hotfix for missing validation (#577) * Updating oscars-backend service to support nullable projectId property * Feedback updates * Updating tests to cover NsiService with new Connection.projectId field * Cleaning up HoldController imports. Updating HoldControllerSteps to include a test with a projectId field when placing a connection hold * Ensuring the NsiService.hold() method only attempts to process the projectId field if it's not null, and not empty * add CORS origin customization property (#581) * fix: [OS-604] yang patch serialization (#582) * fix serialization error * more serialization fixes update spring boot and other libs * add auth interceptor to patch client * remove baseUrl from restClient --------- Co-authored-by: Alejandro Albino <aalbino@lbl.gov> Co-authored-by: Alex Albino <webmaster@alexventure.com> * Adding initial testing for NSI SOAP API (NsiProvider), with and without projectId field * Adding topology service mock data for NsiProvider testing * Adding stubs * make nsiprovider single test pass * Current work sorting out Nsi Provider test steps * Cleaning up NsiProvider cuke testing * Held connections now get cleared per test step in NsiProviderStep * Ensuring a blank projectId field with NSI SOAP API reserve() call results in a null projectId * feat: [OS579] add project id set & view in frontend (#585) * add project id set & view in frontend adjust hold delay minor fix for standalone CORS config * readjust hold delay * Initial work on CI/CD integration of OSCARS Backend profiler solution * Attempt to add mock env var values * Adjust DinD DOCKER_HOST value * Attempt to configure gitlab runner for DinD * Attempt to manually build/run docker containers to perform OSCARS profiling in CI/CD pipeline * Adjusting for resource cleanup with manual docker run (CI/CD OSCARS profiling pipeline) * Removing unused service for backend:profile * Update CI/CD stuff * Attempting dind image * Changing DOCKER env var settings, attempt to get profiler pipeline working * Debugging commit * Attempt to use docker compose * attempting to enable r/w permissions in the /app/profiling directory, oscars-backend * Adjusting DELAY_MIN * Adjusting docker volume config for profiling data * Moving gitlab ci script stuff around * Attempt to manually copy profiling data out of oscars-backend container during pipeline job * Adjusting profiling docker compose up command * Adjusting gitlab ci YAML for profiling * Troubleshooting artifact storage of profiling data * Adding slight delay to allow dind service start up * Adjusting gitlab ci YAML for dind service check * Troubleshooting dind service * Attempting to troubleshoot volume R/W permission issue between host and docker container * fix: [OS-618] [OS-617] [OS-619] multiple improvements (#589) * avoid NPE if if topology ever returns a null port * use new topo-common refactor some string constants & logging * small fixes to pass tests * small logging refactor, use restclient to get service config * don't die if dry run errors out * inject custom object mapper to existing message converter * improve logging, dry run handling * don't retry dry runs * add dry runs to NsoLspStateSyncer * use updated NsoVPLS and NsoLSP from topo-common add tests to exercise equality logic for syncing * fix hop order * detect empty dry runs * organize json test files into directories * replace entire VPLS instance instead of the device containers * fix the paths that i broke * Asserting the expected FILENAME file was actually generated during profiling * Attempt to remove old docker compose network cruft * Only run docker network rm on oscars-local-net if it exists * Remove wrong label property in docker compose profiling network config * It helps if the -f flag doesnt have a space in it... * add some docs to the dockerfile fix a couple misc errors * Attempting to setup CI/CD pipeline for OSCARS backend load testing with Apache JMeter * Fix gitlab-ci.yml * Attempt to troubleshoot docker compose network issue * Helps if we use the correct docker compose YAML config * Split up docker compose between profiling and load test setup * Attempting to fix load test CI/CD for OSCARS backend * Initial scaffolding for additional NSI Reserve Test Suite (NsiProvider) * Current work on checkin NSI SOAP reservation state * Clean up unused imports * Checking the correct mapping property for reservation state, refactoring test * Refactoring NsiService, NsiStateEngine to support callback handlers for reserve() state changes. Refactoring NsiAsyncQueue.processQueue() task into testable components. Updating the NsiProvider cuke feature and testing steps to work with async processQueue() task. Adding new AsyncCallback class to work with async callback handlers. * Updating doc blocks in NsiStateEngine, update NsiProviderSteps to limit story scope (revised AC) * update ESE controller tests to use real OSCARS logic (#594) * fix: [OS-634] make sure NSO dismantles happen before builds / redeploys (#593) * make sure NSO dismantles happen before builds / redeploys * add tests around queue preprocessing * feat: [OS-641] projectId now a set instead of a single string (#595) * projectId now a set instead of a single string * projectId now projectIds plural * update ESE controller tests to use real OSCARS logic (#594) * projectId now a set instead of a single string * projectId now projectIds plural * correct has / does not have a projectId checks * 1.2.33 release branch back to develop (#597) * make backend:profile an optional job (#598) (the pipeline failed for the release branch since backend:profile has a rule not to run then, but it was a non-optional needs dependency for backend:push) * Oscars annual reporting (#599) * add a python database report tool * fix npe when reservehandler null (#601) * more npe fixing (#602) * feat: [OS-633] [OS-632] add projectIds field validation against ESDB graphql data (#600) * add projectIds field validation against ESDB graphql data * add orcid check digit validation * Fix sb queue (#603) * fix southbound queue behavior * fix: [OS-652][OS-653] multithreading performance, and DB error fixes (#605) * Fix database error when committing a modify (OS-652) Add ScheduledTaskConfig bean to tell the Spring TaskScheduler to use more than one thread Rename `nso.queue-interval-millisec` to `nsi.queue-delay` * add sb.trigger-delay to control southbound trigger frequency * separate slow NSI housekeeping from other scheduled threads add resv.state-delay and nsi.housekeeping-delay props * fix NSI provider tests * fix: [OS-657] fix rare NPE during commit modify (#606) * Handle rare NPE during a modify commit Ensure triggerSouthbound() finally releases connection lock Fix vlan validation Clean up some logging Don't throw exception when receiving multiple terminate requests, just succeed * fix determining port encapsulation * update esnet topology document * more gracefully handle errors * update changelog, standalone topology (#608) * [OS-659] add l2vpn release operation (#613) * add l2vpn release operation * add ese to local cors * feat: [OS- 661] list sorting (#614) * add sort params to ConnectionFilter * fix reverse sorting error * fix tags and vlan id comparators, adjust sortProperty json representation * handle null values * fix for missing nsovcid * renamea couple sortproperties * check for connection state before trying to pull operational state from nso (#616) * fix: [OS-672] [ESE-361] Fix last updated time, modify failure when vlan id changes (#622) * correct L2VPN last modified * fix modify error when VLAN changed * feat:[OS- 670] add support for strict policing to NSI (#623) * correct L2VPN last modified * fix modify error when VLAN changed * add a policing parameter for NSI * update test * update test _correctly_ * add a policing parameter for NSI * update test * update test _correctly_ * feat: [OS-647] add ESDB data sync for bandwidth utilization (#624) * add ESDB data sync for bandwidth utilization * forgot to add test suite files * fix: [OS-675] Optimize bwutil sync (#626) * optimize bwutil sync * fix id for removal * properly fix ids for removal * add some policing nsi logging * adjust some logging * return policing tvt * add makefile for mirroring DB from prod (#627) * perf: [OS-680] Add connection list caching (#629) * try to support NSO 6.7 * add a feature flag for the params encoding * add caching to the connection list * fix phase filtering * add to changelog * update java images * add default values for sort property and sorting to help caching * improve phase filtering * fix yang dry run uri target * add a bit of logging to yang patch dry run * update pom xml (#630) --------- Co-authored-by: Alejandro Albino <aalbino@lbl.gov> Co-authored-by: Alex Albino <webmaster@alexventure.com> --------- Co-authored-by: Alejandro Albino <aalbino@lbl.gov> Co-authored-by: Alex Albino <webmaster@alexventure.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.